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Abstract 

A scheme for recognizing 3 D objects fromsingle 2D inages is introduced. The schena proceeds in two 
stages. In the first stage, the categorization stage, the inage is conpared to prototype objects. For 
each prototype, the viewthat rust resenbles the inage is recovered, and, if the viewis found to be 
similar to the inage, the class identity of the object is determined. Inthe secondstage, the i dent i ficat i on 
stage, the observed obj ect is conpared to the individual rodels of its class, where classes are expectedto 
contain obj ects with relatively si ml ar shapes. lor each rodel, a viewthat natches the inage is sought. 

If such a viewis found, the object’s specific identity is determned. The advantage of categorizing the 
object before it is identified is twofold. First, the inage is conpared to a snaller number of rodels, 
since only rodels that belong to the object’s class need to be considered. Second, the cost of comparing 
the inage to each rodel in a class is very low; because correspondence is computed once for the whole 
class. More sped Really, the correspondence and obj ect pose computed in the categori zati on stage to align 
the prototype with the inage are reused in the identification stage to align the individual rodels wth 
the inage. A a result, identification is reduced to a series of simple template comparisons. The paper 
concludes with an algorithmfor constructing optinal prototypes for classes of objects. 
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1 Introduction 

Our w)rl d contai ns an overwhel ling vari ety of obj ects. 
While people demonstrate outstanding abilities to mm 
orize and recognize thousands of objects [27, 37, 38], 
computer vision applications largely fail to acconra- 
date these numbers. Apparently, the rain tool that en¬ 
ables people to effectively handle this rassive arount 
of obj ects i s categori zati on. By di vi di ng the obj ects i nto 
classes, the visual systemis capable of concluding prop¬ 
erties of unfamiliar objects fromtheir resenblance to 
f anil i ar ones. lor faniliar obj ects, categorization offers 
an indexing tool into the stored library of object repre¬ 
sent ati ons. 

Recognition can be performed in different “levels of 
abstraction”. lor example, the sana object can be rec¬ 
ognized as a face, a human face, or as a specific person’s 
face. Psychological studies suggest the existence of apre- 
f erred level for recognition, called “the basic level of ab¬ 
straction” [33] . Existing computational schemas usually 
approach recognition in ei ther one of tv» levels. Several 
schemas attempt to cl assify obj ects in their basic level 
of abstraction (we refer to this task by categorization), 
while other schemas attempt to determine the specific 
identity of objects (we refer to this task by i dentifica- 
tion). This paper presents a novel approach for recogni¬ 
tion that conbines the two tasks. 

T>see howthe two tasks are related, consider the fol- 
1 ova ng exanpl e. Suppose you are wal ki ng down a street, 
and sonaone is coning towards you. You look at the 
person’s face, and it looks faniliar, but you cannot tell 
whoit is. So youtry to pi cture the people youknowwho 
1 ooklike the personyousee, until finally, yourealize who 
the person is. 

Anunber of hypotheses can be drawnfromthi s story. 

First, recognition can be broken into two stages: cat¬ 
egorization and identification, where categorization is 
believed to precede identification. Second, during the 
course of recognition the image is compared against a 
number of object models. Asuning that i ndeed catego¬ 
ri zati on precedes identification, only models that belong 
tothe object’s class needtobe considered. Finally, when 
a new model is compared to the image, the comparison 
process raybenefit frommthe use of information acquired 
duri ng categori zati on. Note that the si tuati on descri bed 
here is not specific to faces. Che can imagine that simi¬ 
lar situations occur when other objects, such as animals, 
cars, and chairs, are observed. 

T> see how inf or rat ion acquired during categoriza¬ 
tion can be usedfor identification, consider the example 
of face recognition. Wenaface is recognized, the image 
positions of its parts and features are known. In partic¬ 
ular, an observer already knows where the eyes, nose, 
and mouth are and can even infer the direction of gaze 
and expression. The person’s identity is not essential for 
extracting and 1 ocating these features. Instead, they are 
matched against features in a “generic” representation. 

In addition, other features, such as a beard, hair style, 
and wrinkles, that ray better distinguish between dif¬ 
ferent persons maybe located. Mre generally, we can 
postulate that, during categorization, sub-structures of 


the objects (such as parts and features) are extracted 
and located with respect to a generic model, and the 
object’s pose is determined. 

Tbfollowthis example, I propose a schema for recog¬ 
nizing 3 D objects fromnsingle 2D views that conbines 
the two stages, categorization and identification. Cat¬ 
egorization is achieved by aligning the image to proto¬ 
type objects. The prototype that appears most similar 
tothe image determines the class identity of the object. 
After the object is categorized, its specific identity is de¬ 
termined by aligning the observed object to individual 
models of its class. % first categorizing the object, not 
only the number of models considered for identification 
is reduced, but also the cost of comparing each model 
tothe image si gnificantl y decreases. This is achieved by 
reusing the correspondence and pose computed for the 
prototype in the categorization stage to align the image 
vaththe individual models. Wshowinthis paper that, 
albeit a perfect natch between the prototype and the 
image is not obtainable, the correspondence and pose 
can be computed for the prototype, and can be used 
to bring the image and the object’s model into align¬ 
ment. (Jnsequently, recovering the correspondence and 
pose for the individual models becomes unnecessary, and 
identification is reduced to a series of simple template 
conpiari sons. 

The rest of this paper is divided as follows. Section 2 
reviews the rain existing approaches for categorization 
and identification. Section 3 presents the schema of 
recognition by prototypes. Section 4 proposes analgo- 
rithrnfor generating optimal prototypes for the schema. 
Section 5 discusses the relevance of the schema to hu¬ 
man recognition. Implementation results are presented 

1 n Secti on 6. 

2 Previous Approaches 

Existing schemes for categorization often use a “reduc¬ 
tionist” approach. The image, which contai ns a detailed 
appearance of an object, is transformed into a compact 
representation that is invariant for all objects of the 
same cl ass. Che commem approach to generati ng such a 
representation is by decomposing the object into parts. 
Fhrts are extracted by cutting the object in concavities 
[17, 22, 43] and 1 abel ed accordi ng to thei r general shape. 
The labels, together with the spatial relationships be¬ 
tween the parts, are used to i dentify the class of the ob¬ 
ject [4, 6, 7, 26] . Asecond approach extracts the parts of 
the object that fulfill certainfunctions. The list of func¬ 
tions is used to determine the object’s class [16, 39, 47]. 

Schemas that break obj ects into parts are insufficient 
to explain all the aspects of recognition for the foil ova ng 
reasons. First, in many cases objects that belong to the 
sana class differ only by their detailed shape, while they 
share roughly the sana set of parts. Mreover, even ob¬ 
jects that at soma level maybe considered belonging to 
different classes, such as a cat and a dog, ray also share 
roughly the same set of parts. To solve this problemnsev- 
eral system also store, in addi ti on to the part structure 
of the objects, the detailed shape of the parts [2, 6, 7]. 
Aiother problemnis that many of the techniques for rec¬ 
ognizing objects by part decomposition rely on finding 
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the entire parts fromthe inage. 

T> recognize the specific identity of objects, a rel¬ 
atively detailed representation of the object’s shape is 
conpared wth the inage. Ai example for such nath- 
ods is alignnant [3, 9, 12, 13, 18, 25, 40, 41]. Aignnant 
involves recoveri ng the position and orientation(pose) in 
whi ch the obj ect i s observed and conpari ng the appear¬ 
ance of the object fromthat pose with the inage. Oily 
a fewattenpts have been nade in the past to extend 
the alignnant schena to the problemof object catego¬ 
rization (e. g., [36]). The nain diffiulty in applying the 
alignnant approach is the recovery of the pose of the 
observed obj ect. In rost inplenantations this involves 
a tina-consuming stage for finding the correspondence 
between the rodel and the inage. The process beconas 
impractical when the inage is conpared against a large 
library of objects, because typically the correspondence 
is established bet ween the inage and each of the rodels 
in the library separately. 

T> handl e 1 arge 1 ibraries, indexi ng nathods were pro¬ 
posed (e.g., [20, 46, 14]). The basic idea is the following. 
Acertainfunction is defined and applied to the views of 
all the objects in the library. The object rodels are ar- 
rangedin a look-up table indexed by the obtained func¬ 
tion values. Wen an inage is given, the function is 
appl i ed to the i nage, and the obt ai ned val ue i s used to 
index into the table. To reduce the size of the table and 
the complexity of its preparation, invariant functions, 
functions that when applied to different views of an ob¬ 
ject return the sana value regardless of viewpoint, often 
are used as the i ndexi ng functi ons. 

Indexing rot hods suffer from several short comngs. 
First, existing indexing nathods handle only rigid ob¬ 
jects. Extending these nathods to handle classes of ob¬ 
jects has not been discussed. Second, because of com 
pi exity issues, i ndexi ng f unct i ons usually are applied to 
snail numbers of features. A a result, high rates of 
false positives are obtained, and the effectiveness of the 
i ndexi ng i s reduced. 

The schena presented in this paper is designed to 
work where traditional approaches to categorization and 
i ndexi ng f ai 1. The s chena conbi nes bot h cat egor i z at i on 
and identification of objects, and uses fairly detailed rep¬ 
resentations for objects. Ehther than indexing directly 
to the specific object rodel, the schena indexes into 
the library of objects by categorizing the object. The 
classes handled by the schena include objects with rel¬ 
atively si ml ar shapes. To fit into the schena, insona 
cases basic level classes are broken i nto sub-cl asses. The 
general problemof categorization therefore nay require 
additional tools. 

3 Recogli ti on by Phototypes 

The recognition by prototypes schena proceeds as fol¬ 
lows. Alibraryof 3 D object rodels is stored in mm 
ory. The rodels in the library are divided into classes, 
and 3 D prototype objects are selected to represent the 
classes. lor every class, the rodels in the class are 
aligned in the library with the prototype object. The 
role of this 3 D alignnant will becona clear shortly. 


A recognition tina, an incoming 2D inage is first 
natched against all of the prototypes. For each proto¬ 
type object, the systemattenpts to recover the viewof 
the prototype that rost resenbles the inage. To do so, 
the systemrecovers the correspondence betweenthe pro¬ 
totype and the i nage, and, using this correspondence, it 
determines the transfornation that best aligns the pro¬ 
totype wth the inage. This trans for nation, referred 
to as the prot ot ype transform, is then applied to the 
prototype, and the si nil arity between the transfornad 
prototype and the actual inage is evaluated. Since the 
observed obj ect in general differs fromthe prototype ob¬ 
ject, aperfect natch between the two i s not anticipated. 

The systemtherefore seeks a prototype that reasonably 
natches the inage. Qice such a prototype is found, the 
class identity of the object is determined. 

Ater the object’s class is determined, the systemat¬ 
tenpts to recover the specific identity of the object. A 
this stage, the inage is natched against all the rodels 
of the object’s class. Ear each of these rodels, the sys- 
temseeks to recover the transformation that aligns the 
rodel withthe inage. Awll be shownbel ow; since the 
rodels are aligned in the library wth the prototype, the 
transfornati on that best aligns the prototype wththe 
inage is identical to the transfornati on that aligns the 
rodel to the inage. The prototype transformtherefore 
is applied to the specific rodels, and their appearance 
fromthis pose is conpared wth the inage. The rodel 
that aligns with the inage, if there is such, determines 
the specific identity of the object. 

The rest of this section is divided as follow. In Sec¬ 
tions. 1 the object representation used in our schena is 
presented. Section3. 2 describes the categorizationstage, 
and Section 3. 3 describes the i dentificati on stage. 

3.1 Object representation the linear 
combi nat i on s chena 

In our schena, an object is rodel ed by a natrix M 
of size n x k, where n is the number of feature points, 
and k represents the degrees of freedomof the object. 

A vector a £ TZ k , referred to as the transf ormvect or , 
represents the transformation applied to the object in a 
certain view; and the object’s appearance fromthis view 
i s gi ven by 

1 = M~a (1) 

In the rest of this section we explain the use of this nota¬ 
tion. The notation follow fromthe linear combination 
schena [42], which is bri efly revi ewed bel ow 

Under the linear combination schena an object is 
rodel ed by a snail set of view, each is represented 
by a vector containing point positions, where the points 
in these view are ordered in correspondence. Navel 
vi ew of the obj ect are obtai ned by appl yi ng 1 i near com 
binations to the stored view. Aiditional constraints 
nay apply to the coeffiients of this linear combination, 
(imputing the object pose therefore requires recovering 
t he coeffii ent s of t he 1 i near conbi nat i on t hat al i gn t he 
rodel with the inage and verifying that the recovered 
coeffiients indeed satisfy the constraints. The nathod 
handles rigid objects under weak-perspective projection 
(nanaly, orthographic proj ection foil owed by a uniform 


2 



( 6 ) 


scaling). It was extendedto approxirate the appearance 
of objects withsroothbounding surfaces and to handle 
articul ated obj ects. In our representation, the columns 
of the rodel natrix M contain view of the object, and 
t he coeffii ent s of t he 1 i near conbi nat i on t hat al i gn t he 
rodel with the inage are given by the transformvector 
~a . 

R>r concreteness, we reviewthe linear combination 
schem for rigid objects, (insider a 3 D object O that 
contains n feature points (A, Ij, $), 1 < i < n . Ihder 
weak-perspective projection, the position of the object 
following a rotation R, translati ant, and scaling s is 
gi ven by 


Xi = s r n Xi + s ^ 2 Yi +s r i3 Zi +t x ^ 

Vi = s r 2 iXi +.s r 22 Yi +.s r 23 Zi +t y ^ ' 

where r ij are the components of the rotation natrix, R, 
andf x , ty are the horizontal and vertical components of 
the transl ati on vector, t respectively. 

Denote by X,Y ,Z , ~x , ~y n (ye®ors of Xi, Y, Y ¥ 
and y values respectively, and denote 1 =(1, .. . , 1) 
lZ n , we canrewite Eq. 2 i n a vector equati on as follow: 


where 


Therefore 


~x 

= ql +a 2 Y 

-\-a 

3 Z +a 4 I 

~y 

= hXM'jY 

+b 3 
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ai 
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= s r 2 1 

a 2 

= s r 12 

b 2 

= s r 22 
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= S 7*13 

b 3 

= s r 23 
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— I X 

h 

= t y 


~x , ~y £ s|)¥{Z,l} 


(3) 


(4) 


Efferent views of the object are obtained by chang¬ 
ing the rot ati on, scale, and translation parameters, and 
these changes result in changing the coeffii ents inEq. 3. 
Wnay therefore conclude that all the views of a rigid 
object are contained in a 4D linear space. 

This property, that the view of a rigid object are 
contained in a 4D linear space, provides a method for 
constructing viewer-centeredrepresentations for the ob- 
j ect. The i dea i s to use i nages of the obj ect to construct 
a basi s for thi s space. In general, two vi ew provi de suf¬ 
ficiently many vectors. Therefore, any novel viewis a 
1 i near conbi nat i on of t wo vi ew [30, 42] . 

N)t every linear conbi nation is avalidviewof a rigid 
obj ect. Eol 1 owi ng the orthonornali ty of the rowvectors 
of the rotation natrix, the coeffiients in Eq. 3 moist 
satisfy the two quadratic constraints 

a\ +al+al=bl+bl+bl 

cq&i +a 2 b 2 +a 3 b 3 =0 ^ ' 


Wen the constraints are not satisfied, distorted (by 
stretch or shear) pictures of the objects are generated. 
Incase a viewer-centeredrepresentationis used, the con¬ 
straints change in accordance wththe selected basis. A 
third view of the object can be used to recover the new 
constraints. 

R)r the purpose of thi s paper a rodel for a ri gi dobj ect 
can be constructed by bui 1 di ng the fol 1 owi ng n x 4 rodel 
natrix 

M=( X,Y ,Z, 1) 


View of the object can be constructed as follow 


~x = M~ci 

~y = M) 


where ~a =(a 1; §, §, f) and b =(&l, i, J>, £) are the 
coeffiients fromEq. 3. Notice that the two linear sys¬ 
tems can be merged i nto one by constructi ng a rodified 
rodel natrix in the following way 


~x \ _ M 0 W a 

y) ~ l o m [b 


(7) 


Similar constructions can be obtained for objects with 
smooth bounding surfaces and for articulated objects. 

The width of M, k , should then be rodified according 
to the degrees of freedomof the rodeled obj ect. A was 
mentioned above, viewer-centeredrepresentations canbe 
obtained by constructing a basis for the AD space from 
inages of the object. Therefore, vi ewer-centered rodel s 
can be obtained by replacing the column vectors of M 
Gwththe constructed basis. 

To summarize, following the linear combination 
schem we can represent an obj ect by a natrix M and 
construct view of the object by applying it to trans- 
formvectors ~a . lor rigid objects not every transform 
vector is valid; the components of the transformvector 
must satisfy the two quadratic constraints. Efecognition 
invol ves recovering the transformvector ~a and verifying 
that its components satisfy the two constraints. Ignor¬ 
ing these constraints will result inrecogni zing the obj ect 
evenwhenit undergoes general 3 D affie transformation. 

In the analysis below we largely ignore the quadratic 
constraints. These constraints, however, can be verified 
both during the categorization stage as well as during 
the identification stage. 

3.2 Cat egori zat i on 

The recognition by prototypes schem begins by deter¬ 
mining the object’s category. This is achieved by com 
paring the observed obj ect to prototype objects, objects 
that are “typical exemplars” for their classes. R)r a given 
prototype, the viewof the prototype that most resem 
bles the inage is recovered and compared to the actual 
inage, and the result of this comparison determines the 
class identity of the object. 

Wbegin our description of the categorization stage 
by defining the data structures used by the schem. A 
class C =(P, {MM, • • ;},) M a pair that includes a 
prototype P and a set of object rodels M i, M, ■ ■ I- , 
Both the prototype and the rodel s are represented by 
n x k matrices, where n defines the number of feature 
points considered, and A; denotes the degrees of freedom 
of the objects. For the sake of simplicity we assum here 
that all the objects in the class share the sara number 
of feature points, n , and that they have similar degrees 
of freedorp k . N)te that similar objects tend to have 
similar degrees of freedom(e. g., all of themare rigid). 
Both assumptions are not strict, however. The schem 
canbe rodified to tolerate both varying number of fea¬ 
ture points as well as different degrees of freedom The 
details will be discussed 1 ater in this paper. N)te that 
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the obj ects can be rodel ed by ei ther obj ect- centered or 
viewer-centeredrepresentations. Incase viewer-centered 
representations are used we shall assure that the rodels 
represent the objects fromthe sana range of viewpoints. 

Aclass in our schena contains objects withsimlar 
shapes. These objects share roughly the sana topolo¬ 
gies, and there exists a “natural” correspondence be¬ 
tween them (insider, for instance, the two chairs in 
Figure 1. A though the shapes of these chairs are dif¬ 
ferent, and som parts (e.g., the arm) appear only in 
one chair and not inthe other, anatural correspondence 
between features inthe two objects can be determned. 

Inthe library of rodels, the natural correspondence 
between obj ects is nade explicit. It is sped Red by the 
order of the rowvectors of the rodels. Specifically, given 
a prototype P and obj ect rodels Mi, . . ;, ,wef£>rder 
the rows of these rodels such that the first feature point 
of P corresponds to the first feature point of each of the 
rodels Mi, . . i, ,anifso forth. 

Giventhe library of objects and gi ven an i ncomng i m 
age, the recognition by prototypes schena begins by cat¬ 
egorizing the object observed in the inage. To achieve 
this goal, the prototype objects are aligned and com 
pared to the inage. R>r every prototype, the correspon¬ 
dence between the inage and the prototype is first re¬ 
solved, and, using this correspondence, the nearest pro¬ 
totype viewis recovered. % doing so, the schena de¬ 
couples the two factors that affect the appearance of the 
object inthe inage, namly, viewvari ations and shape 
variations. % selecting the nearest prototype viewto 
the inage, the schena conpensates for viewvariations. 
Then, by evaluating the simlarity between the nearest 
prototype viewandthe actual inage, it accounts for the 
differences in shape between the prototype and the ob¬ 
served obj ect. 

The first stage innatching the prototype to the i nage 
involves the recovery of correspondence between proto¬ 
type and inage features. In existing system for rec¬ 
ognizing the specific identity of objects establishing the 
correspondence between inages and object rodels in¬ 
volves a tina-comumng process in which sophisticated 
algorithm are applied [10, 13, 15, 18, 23, 25, 35, 41]. 
These algorithm rely on the property that, when the 
correct correspondence between a rodel and an inage is 
established, a near-perfect natch bet ween the two is ob¬ 
tained. \ffile this assunptionis validfor identification, 
it cannot be used under our schena since the prototype 
and the inage generally represent different objects. 

T> determne the correspondence between the proto¬ 
type and the i nage, we define an obj ecti ve functi on that 
is applied to the prototype and the inage under a given 
correspondence and that obtains its mni nmmunder the 
correct correspondence. The objective functi on vail naa- 
sure the quality of the natch bet ween the prototype and 
the inage. Nmaly, under this naasure the correct cor¬ 
respondence is the one that brings the prototype into 
its best alignnant with the inage. G ven this objective 
function, correspondence is aconbinatorial optimization 
probl erp and so mni mzati on techni ques can be used to 
resolve the correspondence between the prototype and 
the inage. This paper does not propose a specific tech¬ 


nique for solving the correspondence problem 

Asumng the correspondence probl emcan be solved, 
the schena proceeds as follows. Gven a prototype P 
and an inage I, we generate a view vector ~v fromthe 
inage by extracting the location of feature points and 
arranging themin a vector. The points in ~v are ordered 
in correspondence to the prototype points; that is, the 
first point in ~v corresponds to the first point in P and 
so forth. The prot ot ype t ransf ormis the transfornation 
that brings the prototype points as close as possible to 
their corresponding inage points. The prototype trans¬ 
form therefore, is the transformvector b that mni nazes 
the Enelidean distance betweenthe prototype andinage 
poi nt s, nanal y 

niin || Pb' — ~v || (8) 

¥ 

A solution for (8) is obtained as follows. Asumng P 
is over determned; that is, P is n x k where n > k and 
r a rik (P) = k , and denote by iP = (P T P )~ 1 P T the 
pseudo-inverse of P , the prototype transform b , is given 

by 

b =P + ~v ( 9 ) 

and the nearest prot ot ype mew ~p is obtai ned by appl yi ng 
P to the prototype transform b , that is 

~p =P> =P P^~v (10) 


The nearest prototype viewis nowconpared to the 
inage, and their resenblance determines the class iden- 
tityof the object. The qualityof the natch between the 
prototype and the inage is defined by 

D(P, T;)=||P || =1KW || (11) 

To eliminate effects due to scaling of the object, this 
naasure should be nornalized, as is illustrated by the 
exanple below (insider an obj ect seenfromsom view 

Its distance to the prototype is given by D(P, i)l 
Suppose the object is nowseen froma newview ~v 2 that 
is identical to f y except that the object is nowas twee 
as close to the canara. Ihder these conditions ~v 2 =2”L>i, 
and its distance to the prototype i s gi ven by D(P, = 

2 D(P, ij). Gearly, we should have a naasure that is 
independent of the distance of the obj ect to the canara. 

Qie way to obtain such a naasure is by dividing D(P, ~v ) 
by the norm|| ~v || 


D(P, 


\\(P1*-I)v 

’ in> 11 


( 12 ) 


D(P, ”1)18 proposed here as an objective function for 
establishing the correspondence betweenthe prototype 
and the inage. Inother words, we expect that if the ob- 
j ect bel ongs to the prototype’s cl ass then.D(.P, ~v ) obtains 
i ts mni nal val ue when ~v i s ordered i n correspondence to 
P . Aiy other permit ati on will increase the value of D. 
Fornally, denote by a a per nutation natrix, we as sum 
that 

D(P, ~v ) =mB(P, a ~v ) (13) 

<7 

The naasure D(P, ~v ) has a second role. Since it raa- 
sures the simlarity between the prototype and the im 
age, it can also be used to determne the obj ect’s class. 
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Ai obj ect observed i n a vi ew ~v bel ongs to the cl ass rep¬ 
resented by a prototype P if 

D(P, -v)<e (14) 

for som constant e > 0. Wrefer to (14) as the catego- 
ri zat i on cri t eri on. 

The categorization stage proceeds as follows. Gven 
an inage I and a prototype P , the correspondence be¬ 
tween P and I is resolved by mnimzing the masure 


exanple for a masure that considers both the proxim 
i ty and si ml ari ty between feature points is the foil owng 
masure. Each feature point is associated with a la¬ 
bel (such as a corner or an inflection point). i)gain, the 
masure D(P, ~v ) is applied, but this tim only correspon¬ 
dences between points with simlar labels are allowed; 
namly, corners in the inage can only natch corners in 
the prototype, and, si rid ari y, inflection points can only 
natch inflection points. Gher exanples for masures 


D(P , cr v ) overall possible per nutation a of v , and if t^bg^. CO nbi ne proxi riity and si rid ari ty i nclude masures 
obtai nedrimi mum D(P , v ) is belowthe thresholds , thenthat retainthe tangent or the curvature of points. Mre 


the class identity of the object is deterrimed. 

Nate that in our schem the prototype and the cate¬ 
gorization criterion determne the actual division of ob¬ 
jects to classes; an object belongs to a certain class if 
its views are suffiiently simlar, according to the cate¬ 
gorization criterion, to view of the prototype. Ihder 
the above definition, an obj ect belongs to a prototype’s 
class if the total difference betweenits feature points and 
their corresponding prototype points does not exceeds . 

The masure D(P , ~v ) defined here deterrimes the sim 
ilarity between the prototype P and the view ~v using 
only the distances between feature points. In general, 
since correspondence is diffiult to achieve, such a ma¬ 
sure would not be robust. Including additional inforna- 
tion about the features in the si ml ari ty masure nay 
increase the robustness of the schem. A so, masures 
that consider only the proxi mty of feature points are 
1 i ridedin term of di vi ding the li brary into cl asses, since 
they i nduce cl asses of obj ects w th hi ghly si ml ar shapes. 
Masures that consider additional i nf or nation can ex- 
tendthe classes to include larger sets of objects. 

The masure D(P, ~v ) can be enriched by considering 
the siridarity between corresponding points. Asinple 


sophisticated masures nay conpare the topologies of 
the objects in the two view, or, in other words, verify 
that the objects share simlar part structures in 2 D. 

A useful technique in masuring the simlarity be¬ 
tween the inage and the nearest prototype view is to 
consider a different set of features than the set used to 
determne the prototype transform The rational behind 
this technique is that it is generally diffiult to recover 
exact feature-to-feature correspondence, and while such 
correspondences are necessary for recoveri ng the proto¬ 
type transforrp simlarity masures can be successfully 
applied even in the absence of exact feature-to-feature 
correspondence. This idea resenbles the basic principle 
of thealignmnt algorithm) 18, 41], in whi ch a snal 1 set 
of points is used to conpute the object pose, while a 
larger set of points is used to verify this pose. 

It should be noted that the general flowof the schem 
and, in particular, the i dent i Heat ion stage are indepen¬ 
dent of the specific choice of si ml ari ty masure. A has 
been noted above, the masure affects the division of 
rodel libraries into classes and the selection of optinal 
prototypes for these classes. Ai exanple for selecting 
the optinal prototype for a given class under the ma- 
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sure specified in (12) (for either 1 abeledor uni abeledfea¬ 
tures) is described in Section 4. 

Fi nal 1 y, al though the nai n obj ecti ve of the categori za- 
tionstage is todetermne the class identity of the object, 
the categori zati on schena descri bed above is useful even 
if the object’s category cannot be determined. Section 
3.3 belowshows that the prototype transformcan be 
reusedto alignthe inage with the specific rodels. Ghn- 
sequently, following the categorization stage the cost of 
conparing the inage to each of the specific rodels is 
substantially reduced since the diffiult part of recover¬ 
ing the transfornation that relates the rodels to the 
inage is applied only to the prototype objects. A a 
result, if the class identity of the object cannot be deter¬ 
mined we still need to consi der all the speciHe rodels in 
the library, but the overall cost of conparing the rod¬ 
els to the inage would be lowbecause correspondence is 
conputedonce for the whole class. 

3. 3 Ident i ficat i on 

After the observed object is categorized, the system 
turns to recovering its individual identity. A this stage 
the inage is natched to all the rodels in the object’s 
class. lor each rodel, the systemseeks to recover the 
transfornati on that aligns the rodel to the inage, if 
there is such. In previous schenas this required recover¬ 
ing the correspondence between the inage and each of 
the rodels separately. In our schena, however, this no 
longer is necessary, since the object transformis deter¬ 
mined di recti y fromthe prototype transform Wshow 
in this section that the prototype and the object trans¬ 
form are rel ated by a si npl e transfornati on, whi ch can 
be conputed in advance, and which can in fact be un¬ 
done already in the library of stored rodels. Conse¬ 
quently, the prototype transformcan be reused in the 
i dent i ficat ion stage to alignthe individual rodels wth 
the inage. 

The initial stage of categorization recovers three 
pieces of inf or nation that can be used for identification. 
The three are (i) the object class, (ii) the correspon¬ 
dence between the prototype and the inage, and (iii) 
the prototype transform This infornation is used in 
the identification stage as follows. First, since the ob¬ 
ject’s class is deternimed, only rodels that belong to 
this class are considered. Second, using the correspon¬ 
dence between the prototype and the inage established 
in the categorization stage, and using the stored corre¬ 
spondence betweenthe prototype andthe object rodels, 
the correspondence betweenthe rodels andthe inage 
is inmadiately recovered. Finally, as is shown below; 
the rodel transforrp nanaly, the transfornati on that 
aligns the rodel with the inage, is recovered fromthe 
prototype transform 

Asuib we are given with a view ~v of sona object 
rodel M ;, nanal y 

~v =Af~a (15) 

for sona transformvector ~a . Wen the identification 
process begins, it is still unknowi whi ch of the rodels 
Mi, . . i qf Mie object’s class accounts for the inage 
and what the transformvector ~a is. The first task faced 


by the schena at this stage is to recover the rodel trans- 
forrp ~a . This is done, as is explained below; using the 
prototype transform b =P + ~v defined in (9). Qice ~a is 
recovered, it is applied to all the rodels Af, . . ;, ,a iM 
the rodel for whi ch a near-perfect natch is obtained 
deternimes the object’s identity. 

Theoreml belowestablishes that the rodel transform 
~a can be recovered directly fromthe prototype transform 
b by applying a linear transfornati on whi chi s referred to 
as the prot ot ype-1 o- rrndel t ransf orm Thi s transformhas 
two interesting properties. First, i t is viewindependent; 
nanaly, for anygivenviewof the object, the sana trans- 
formnaps the prototype transformthat corresponds to 
this viewtothe correct rodel transform The prototype- 
to-rodel transformtherefore can be conputed in ad¬ 
vance and stored in the library of rodels. Second, the 
prototype-to-rodel transformcan be used to recover the 
rodel transformregardless of the quality of natch be¬ 
tween the prototype and the inage. In other words, 
even if the prototype aligns poorly with the inage, the 
transfornati on that aligns the rodel with the inage is 
deternimed correctly in this process. 

Theoreml: Given a view 1) = Mi~a . Leb = P + ~v 

be the prototype transform, that is, the transformvec¬ 
tor that best aligns the prototype with the inage. The 
model transform^ "a, can be recovered fromthe prototype 
transfornp , by appl yi ng a imtri y,Anarml y 

~a =j\b 

Ai is referred to as the prot ot ype-1 o- rrndel t ransf orm 
Proof: Notice that 

b =P+A =P~ Mi "a 
iAsum P + Mi is invertible, let 

Ai =(P + Mi) -1 

we obt ai n t hat 

~a =j\b 

□ 

Gbrollary2: The prototype-to-model transformis 

vi ew i ndependent. 

Roof: The prototype-to-rodel transforrp A is in¬ 

dependent of both pose vectors, "a and b . Changing the 
inage ~v wll result inanewpair of pose vectors, ~a and 
b , but siniilar to the old pair, the new pair is related 
through the s am t ransf or mA 8 -. The prototype-to-rodel 
transform/! i therefore can be used to recover the obj ect 
pose for any vi ewof M {. □ 

Ai exists if I*Mi is invertible. This condition is 
equivalent to requiring that the two column spaces of 
P and Mi wll not be orthogonal in any direction. The 
condition holds, in general, when the two objects are 
fairly siniilar. This is illustrated by the following ex- 
anple. (insider the case that both column spaces of 
P and Mi are one-dinansional; nanaly, each represents 
a line through the origin. The only case in this one- 
di nansi onal exanple in which/! does not exist is when 
P and Mi are orthogonal. Eht these lines are farthest 
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apart when they are orthogonal. (Jnsequently, if the 
objects are rel ati vely si ml ar ^4 would exist. 

Since it depends only on the prototype P and the 
rodel M{, the prototype-to-rode 1 transforms! can be 
pre- confuted and stored i n the 1 i brary of rodel s. Every 
rodel M ; £ C is associated wth its own transform/li 
that relates, for every possible view of T/, betweenthe 
prototype transformand the rodel transform To com 
pare the inage to the rodel M ; the rodel transform 
should first be recovered. This is achieved by applying 
Ai to the prototype transformconputed i n the catego¬ 
rization stage. 

41 so, the prototype-to-rodel transforrp A can be 
used to align the rodel M ; wth the prototype P in 3D. 
Ebnote the aligned rodel by M /, M- rodels the sam 
object as M does, since their column vectors span the 
sam space. In addition, the aligned rodel M / has the 
property that it is brought by the prototype transforrp b , 
to a perfect alignnant with the inage. (Jnsequently, if 
the rodels are aligned in the library wth the prototype, 
the prototype transformconputed in the categorization 
stage can be reused for identification with no further 
nanipul ations. This is established in Theorem3 below 

Theorem3: Let M / =M iAi be the rrndel M aligned 
w i th the prototype P . For any vi ew~v ;=hhf the protc 
type transformfor this vibew=P + ~v is identical to 
model transformfor this view, that is, ~fb 

R-oof: Si nee 

M[ =M ^ 

we obt ai n t hat 

M/6 =MiAib =Mi~a =~v 

□ 

Iking Theorem3, the i dentification schem is sim 
plifiedas follows. The rodels Mi, . . j qreMlignedin 
the library with the prototype P by applying the cor¬ 
responding prototype-to-rodel transforrp A i, . . ft, A 
recognition tim, the prototype transform b =P + ~v , is 
applied to the aligned rodels M {, . . /.,/Wording to 
Theorem 1 and 3, by transforming the rodels by b the 
correct rodel, M/, woul d perfectly align wth the i rage. 

In the schem above we assured that full feature-to- 
feature correspondence is established between the proto¬ 
type and the irage. This assunptionis not randatory. 
Mthods for estirating the prototype transformusing 
partial correspondence or by considering other types of 
features (such as line segmnts) can also be used. N)te 
that incase the prototype transformeanonly be approx- 
irated, the accuracy of the rodel transformobtained is 
determined by the qual i ty of thi s approxi rati on and by 
the condition number of the prototype-to-rodel trans¬ 
form/! The condition number of A ; affects the match 
even if Theorem3 is applied, namly, even if the rod¬ 
els are aligned wth the prototype in advance. Conse¬ 
quently, the condition number of the prototype-to-rodel 
transform/! shoul d be taken i nto account when the 1 i- 
braryis divided into classes. 

Finally, the schem can be extended to handle classes 
of objects with different degrees of freedom (insider, 


for instance, the case of similar chairs, som of whichare 
folding. Obviously, the folding chairs have rore degrees 
of freedomthan the regular, rigid chairs, and therefore 
they would be represented in the library by wider ma¬ 
trices than the rigid chairs are. M is expl ai ned bel ow; 
the chairs can be handled in a conron class, and the 
prototype for the class would itself be a folding chair. 

Mre generally, let M 1; . . ; fyeMclass of rodels of 
different wdths, and denote by A; i, . .; .tJieAwi dth of 
Mi, . . i i;es$£ctively. Let P be the prototype for this 
cl ass, and denote by k p the wi dth of P , we set k p to be 

k p =rax{ (16) 

In other words, we require the prototype to have the 
sam degrees of freedomas the rost flexible object in 
the class. Wean set k p accordi ng to our goal since, as it 
is shown in Section 4, the prototype P i s obt ai ned i n our 
schem by manipulating the objects in the class. The 
prototype-to-rodel transform/! is defined in this case 

by 

Ai=(P+Mi)+ (17) 

where A i is Aj, x A;,-. It is straightforwardto extendThe- 
oreml to also include this case, Consequently, for any 
viewof M,-, the rodel transforma can be recovered from 
its correspondi ng prototype transform b by appl yi ng t he 
prototype-to-rodel transform/! to b . N)te that since 
k p > ki the prototype can appear in poses that do not 
natch any possible rodel pose (and therefore in noise¬ 
less conditions they are impossible to obtain). In case 
the object is observed fromsuch a view; A i would rap 
this unmatched prototype transformto the rodel trans- 
formthat corresponds to the nearest ratched prototype 
transform setting k p to be as 1 arge as the naximum 
of ki, . . i .wq aAioid cases where there exist view of the 
object that cannot be accounted for by the prototype. 
Mdel transform that correspond to such view cannot 
be recoveredfromprototype transform. 

3.4 Suimary 

Wpresentedi n t hi s section a schem for recogni zi ng 3 D 
objects fromsingle 2D view that proceeds intwostages, 
categorization and identification. In the categorization 
stage the irage is compared against the stored proto¬ 
types. For every prototype, the correspondence between 
the irage and the prototype is recovered, and the near¬ 
est viewof the prototype is constructed. The similarity 
between this viewandthe irage is evaluated, and, if the 
two are found si nil ar, the cl ass i denti ty of the obj ect i s 
determined. In the i dentificati on stage the observed ob- 
j ect is compared against the rodels of its class. Since 
the prototype andthe rodels were brought inthe library 
into alignmnt, the sam transformation that aligns the 
prototype to the irage also aligns the object rodel to 
the irage. The prototype transformtherefore is applied 
t o t he rodel s, and t he obt ai ned vi ew are compared wth 
the irage. The viewthat is found to be identical up to 
noi se and occl usi on to the i rage determines the i ndi vi d- 
ual identity of the object. 

The presented schem is based on several key princi¬ 
pals. Efecogni ti on is divi ded into two sub-processes, cat- 
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egorization and identification. In both processes rod- 
els are aligned wth the inage, and the identity of the 
object is determined by a 2 D conparison; 3 D recon¬ 
struction of the observed obj ect fromthe inage is not 
performd. The diffiult conponent of the alignnant 
approach, nanaly, the recovery of correspondence and 
object pose, is performd only once for each class; the 
prototype transformis reusedinthe i dent i Heat ion stage 
to ali gn the i nage wth the i ndi vi dual rodels. 

4 Constructirg optimal prototypes 

In the schena above we assured that the classes in the 
library of rodels are represented by prototype objects. 
Since categorization is achieved by natching the ira 
age to prototype objects, the question of howto select 
the best prototype should be addressed. In this section 
we present an algor it hmf or constructing opt inal proto¬ 
types. 

Gven a class of objects, the optinal prototype for 
this class is the object that resenbles the objects of the 
class the rost. Ihder our fornml ation, such an obj ect 
w)uld share as nany features as possible with the ob¬ 
jects of its class, the position of these features on the 
prototype w)uldbe as close as possible to their position 
on the objects, and the prototype-to-rodel transform 
for these objects would be as stable as possible, Efelow 
we showthat the optinal prototype can effectively be 
conputed using principal conponent analysis; that is, 
by conputing the dominant eigenvectors for sona na- 
trix determned by the rodels of the class. 

EH nci pal conponent analysis often is used in clas¬ 
sification probl em to construct classes and prototypes 
[11]. In existing applications, an obj ect is represented by 
a poi nt i n sona hi gh di nansi onal space, where each com 
ponent of this point contains aninvariant attribute of the 
object. Ahyperplane in that space represents a cl ass of 
objects. The goal of the principal conponent analysis 
is, given a set of points (objects), to recover the class 
that these points induce. Qir case is sonawhat differ¬ 
ent. In our case an obj ect is represented by a continuous 
linear space rather than by a point. Wereas the use 
of hyperplanes in other schenas often is arbitrary and 
rade prinarilyfor convenience, their use in our schena 
is appropriate following the linear conbination schena 
[42] (see Section 3.1). 

The differences outlined above also inply differences 
i n t he pr oof t hat pr i nci pi e conponent anal ys i s appl i es 
to our case. Wshowbelowthat the optinal prototype 
can be conputed by pri nci pal conponent anal ysi s. The 
traditional proof needs to be extended since in our case 
objects are represented by continuous spaces rather than 
by discrete points. 

The prototype constructed in this process is a 3 D ob¬ 
ject obtained by nanipulating the objects in its class. 
Toallowthe construction, it seem as if the objects in 
the class should first be brought into alignnant. In par¬ 
ticular, if the objects are represented by vi ewer-centered 
rodels (that is, by sets of their view, see Section 3. 1 for 
details), the different objects woul d then have to be rep¬ 
resented by irages taken fromsimlar viewpoints. Nev¬ 
ertheless, the process presented below does not require 


aninitial alignnant of the objects. The sana prototype 
is obtained in this process evenwhenthe objects are not 
al i gned 

Wnowturn to constructing the optinal prototype. 
First, we define an objective function. Gven a proto¬ 
type P and an obj ect rodel Mi, we define the similarity 
between P and M ; as follow. let ~p be a viewof Mi, 
we naasure the simlarity bet wen the prototype P and 
the viewGj- using (12). Then, we sumthe naasure over 
all possible view of Af. Assuring without loss of gen- 
eralitythat || =1, (14) can be rewitten as 

D(P,iJv=\\(PI* -I )J\ (18) 

Wthout loss of generality, we can assuna that the 
constructed prototype, P, is conposed of orthonornal 
columns. NAe that an over determned natrix P with 
orthonornal columns satisfies P + =P T . Wean there¬ 
fore rewite (18) as 

D(P, iJv=\\(PI?-1)^ (19) 

The distance between/ 3 and the rodel M i is now given 
by summing D(P, ijhover all unit-length (to eliminate 
scaling effects) view of Mi, nanaly 



Tbobtainthe obj ecti ve function, we sumthese distances 
over al 1 rodel s 

E (P)=±[ I \(PF-I)% (21) 

;=i GK||=i 

The object P that minimizes this function is defined to 
be the optinal prototype. 

NAe that (21) is not the only possible objective func¬ 
tion for this purpose. Ai alternative “worst case” ap- 
proachis to naasure the distance betweenthe prototype 
to the farthest rodel inthe class (rather thansuranimg 
this distance over all rodels). Except for being diffiiilt 
to conpute, this naasure also is sensitive to “outlier” 
rodel s. 

The prototype that minimizes (21) can be constructed 
in a process that includes the following steps. 

1 . T> si npl ify the process we assuna the col unm vec¬ 
tors of each of the rodel natrices Mi, (1 < i < /), 
are orthonornal. (In case they are not, we first ap¬ 
pl y a Gam chmdt process to them Sucha process 
obviously does not alter the space of view inplied 
by the rodels.) 

2. Build then xn syranatri c natrix 

F = Y^M l Mj 
8=1 

3. Find the k dominant eigenvectors of F . The opti¬ 
nal natrix/ 3 is constructed fromthese eigenvec¬ 
tors. 
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N)tethat, ingeneral, we are trying to construct apro- 
totype object that w)uld belong to the given cl ass. This 
condition determines the choice of width k for the pro¬ 
totype. If all the rodels share the sana wdth then the 
prototype would assure this wdth. In the rigid case, 
for exanple, k =4 (see Section 3.1). A mntionedin 
Section 3. 3 above, in case the objects have different de¬ 
grees of freedony k is set to be the raxirnmof k i, . .;. 
where k i, . .; .ai;e fche wdths of Mi, . . j r,esjb£cti vely. 
Incase rare than k large eigenvalues are obtained, one 
nay ignore these guideline rules and construct a proto¬ 
type that has higher degrees of freedomthanthe objects 
in the class (see for exanple [31]). 

Theoreml belowestablishes that the algorithmabove 
produces the optical prototype. Wconsider here the 
case that all the objects share si ml ar degrees of freedom 
The sana procedure can be appliedwth slight codifica¬ 
tions to include the case of objects with different degrees 
of freedom 


F therefore is constant for any choice of orthonorral vec¬ 
tors for M, . . „,, arifsoits dominant eigenvectors rep¬ 
resent the best vector space for for any or t honor nal rep¬ 
resentation of the objects. Ghnsequently, P niininilzes 
the objective function regardless of choice of basis for 
the rodels, and therefore it also niininilzes the required 
term 


E ( p ) =E 


(PF-iW 


Tbsurmarize, we showed that given a cl ass of object 
rodels, the optical prototype for this class is given by 
the domnant eigenvectors of the natrix A, wbichis con¬ 
structed fromthe object rodels. Nate that in proving 
Theoreml we showed that the prototype is independent 
of choice of basis for the rodels. This inplies that, in 
order to construct the prototype, the object rodels M 1; 

. .. , iff do not need to first be brought into alignnant. 

The process above guarantees to output the sana pro¬ 
totype object even if the rodels are not aligned. 


Theorem4: Let M i, Mi, . . . , /Me a set of models totype object even if the rodels are not aligned. 

belonging to sons class C . As suns every nmdqliM 

represented by an n x k rmt r i x m t h or t honor nal col um^ R1 e\ance to huTHIl vi si on 
vectors. The prototype P that minimizes the term 

The recognition by prototypes schena uses the general 
.J^ r shape of objects as the cue for recognizing them h was 

E {P) = / y / II {P P ~ I )i|| already nanti one d, classes inour schena containobj ects 

i=i IKII- 1 wth fairly similar shapes. In contrast, the humn vi- 

j , i ■ , , ■ ■ 7 jj .7 ■ i 7 sual systemrecognizes objects using both shape cues as 

where the integration is done over all the uni t -1 engin J & J 4° 1 

-> r u j nj ■ _ j r , , , ■ well as cany other cues, such as color, texture, rotion, 

views t vol each nuclei il#, is composed ol the k eigen- J 


vectors of the matrix 


F = 


that correspond to its k largest eigenvalues. 

Ffcoof: let P be conposed of the k donimant eigen¬ 

vectors of F . ^According to regression principles P mn- 
iniizes the term 


and context, and objects are categorized in their basic 
level of abstraction [33]. Oily little is currently known 
about the underlying processes for recognition used by 
the visual system Bomwhat is known, in spite of the 
differences pointed above, the recognition by prototypes 
schena seem to be consistent in several key issues with 
psychological and physiological findings. In this section 
we bri efly revi ewthese findi ngs. 

The schena presented in thi s paper prorotes the no- 
tionthat categorization and identification are perfornad 


l k 

J2J2\\(pP-I)M 

i=l j=l 

where is the j’th column vector of M{. In other 

words, consider rr\j as a point inlZ n . The space spanned 
by the colunm vectors of P is the nearest k -dinansional 
hyperplane to these points, rriij. The rest of this proof 
extends the claimfromthe discrete sumover the col unm 
vectors of Mi to the continuous integral over all views 
spanned by these vectors, ifecordi ng to our assumptions, 
each natrix M ; contains an or t honor nal set of col unm 
vectors. ffepl aci ng these vectors by another orthonornal 
basis for Af wll not change the natrix P ; that is, P is 
independent of the choice of orthonornal basis for the 
rodels. This is illustrated by the following derivation. 
To obtai n a newort honor nal basis for the colunmspace 
of Mi we can apply a k x k rotation natrix R to Mi 
(nanaly, MiR). P is the best vector space for the new 
set as well, since 

MiR{MiR) T =M iRR T M? =M d Jlf =M 


using similar tools. In both cases view variations first 
are compensated for, and then a viewof either the hy¬ 
pothesized prototype or object rodel is compared wth 
the inage. This is in contrast to rot hods (such as part 
deconposition and functional description) that in gen¬ 
eral handl e ei t her cat egor i z at i on or i dent i ficat i on, but 
do not extend to deal wth both problem. The avail¬ 
able studies in this case are inconclusive. Sona evidence 
seemto i ndi cate that the two processes are handled sepa¬ 
rately by the visual system i)gnosti c and pros opagnosti c 
pati ents often deronstrate degraded i denti ficati on abi 1 i - 
ties, whereas their performance incategorizationrenains 
intact. Lbuble dissociation between the two processes, 
however, has not beenfound, andsothe as sumption that 
the two processes are handl edseparately inthe brainhas 
not been established. In fact, both cells that respond 
to general faces as well as cells that respond to specific 
faces where found lying side by side within the sana 
brain area, STS, of the nacaque ronkey [29] . The vul¬ 
nerability of the i denti ficati on process to brai n 1 essi ons 
can be explained by that the process requires a relatively 
large narory to encode the detailed shapes of objects as 
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wel 1 as sophisti catedi rage processi ng mchani sm to re¬ 
cover a detailed description of the observed obj ect from 
the i nage (see e. g. , [ 19]). 

Aiother idea proposed here is that categorization in¬ 
volves tw) stages: astage of conpensating for viewvari- 
ations folloved by a stage of 2 D conparison to account 
for shape differences. Adecoupling of view variation 
and senantic categorization was suggested by Iissauer 
[24]. Wrrington and Taylor [44, 45] found that pa¬ 
tients that suffer fromlessions in the posterior lobe of 
the right hemsphere deronstrate diffiulties in catego¬ 
rizing obj ects fromunconventional view, whereas their 
perfornance in categorization of objects fromconven- 
tional view renains intact. Aiditional evidence for the 
effect of view variations on categorization perfornance 
were found for heal thy subj ects. Subjects that are asked 
to nan objects respond slower when the objects ap¬ 
pear i n unconvent i onal view [28] . A so, rental rotation 
effects, nanaly, response tire that grows linearly with 
the tilt of the object, were observed in naniing tasks of 
natural objects [21]. 

Finally, the process of categorization presented here 
is achieved by conparing the inage to prototype ob¬ 
jects, and these prototype objects can be constructed by 
nanipulating the famliar objects of the class, ffecent 
studies indicate that response tina in naniing tasks is 
typically shorter and error rates are lower when the ob¬ 
served obj ect is siniilar to the prototype [5]. Simlarly, 
shorter reactiontina is obtained when subj ects are asked 
to answer questions of the type “does the object X be¬ 
long to the class Y! v [34]. Gher studies reported that 
children learn good exanples of classes before they learn 
poor ones [1, 32] and that subjects recall having seen 
the prototype or average configuration of studied face 
inages evenif this configuration was not studied [8] . 

T> sunmarize, although the presented schena gen¬ 
erally does not recognize objects in their basic level of 
abstraction, it is consistent wth psychological and phys¬ 
iological findings in several key issues including a single 
approach for the two sub-problem of recognition, cat¬ 
egorization and i dentihcati on, vi ew dependency of the 
two sub-processes, and the role of prototypes in catego¬ 
rization. The findings discussed here obviously are in¬ 
conclusive, since psychological and physiological studies 
including the ones discussed here have rare than one 
possible interpretation. 

6 Irplenantation 

Totest the ideas presented in the paper, we have inple- 
nanted the schena and applied it to several objects. In 
our inplenantation, the library of rodels includedtwo 
classes. The first (Figure 2) contained two four-legged 
chairs (denoted by Aand B), and the second (Figure 3) 
i ncl uded t wo car rodel s, a VWhnd a Saab. 

T> deronstrate categorization, we used chair Aas a 
prototype andnatchedit to an inage of chair B Corre¬ 
spondences between the prototype and the inage were 
picked nanually, and, using these correspondences, the 
prototype transformwas recovered and applied to the 
prototype. The results of natchingthe transfornadpro¬ 
totype wththe inage are seeninFigure 4. It can be seen 


that the transfornad prototype (riddle figure) assured 
the sana orientation as the observed obj ect (left figure), 
and that the natch bet ween the two is good considering 
that the objects have different shapes. Nate that in this 
inplenantation we allowed the obj ects to undergo gen¬ 
eral ahhe trails for nations in3G, including stretch and 
shear, and so the natch between the prototype and the 
inage was better thanif only rigid transfornati ons were 
allowed. Aiditional exanples using chair Band the two 
cars as the prototypes are shown in Figures 5-7. 

InFigures 8-9 we triedto natchthe prototypes to the 
inages wthwrong correspondences. The results of these 
natches were significantly worse than when the correct 
natches were used. This is consistent with the ideadis- 
cussed in Section 3. 2 that the quality of the natch can 
be used as the obj ecti ve functionfor resol vi ng the correct 
correspondence. 

Figure 10 shows the results of natching a prototype 
four-legged chair to a single-legged offie chair. It can 
be seen that the upper portions of the chairs natch rel¬ 
atively well, while the legs of the chairs do not find ap- 
pr opr i at e nat ches. 

Figure 11 shows the result of natching a prototype 
chair to an inage of a Saab car. A an anecdotal ex- 
anple, we natchedthe hole belowthe back of the chair 
to the windshield of the car and the seat to the hood. 

In general, whatever correspondence is used, the two ob- 
jects woul d natch poorly relative to natching the pro¬ 
totypes to obj ects of their class. 

Figures 12-13 deronstrate the i dentihcationstage. In 
the library we first aligned the rodel for chair Awth 
the prototype chair (chair B) using the prototype-to- 
rodel transform Then, an inage of chair A was cate¬ 
gorized (Figure 5) bynatchingit tothe prototype chair, 
and the prototype transformwas conputed. In the next 
step, the prototype transformwas appliedto the specific 
rodel of chair A The result of this application is seen 
in Figure 12. It can be seen that a near-perfect align- 
nant was achieved in this process. Asimlar process was 
applied to the VWcar in Figure 13 using the Saab car 
as the prototype. (The result of the corresponding cat¬ 
egorization stage is shown in Figure 6.) These figures 
deronstrate that although a perfect natch between the 
prototype and the i nage coul d not be obtai ned, the pro¬ 
totype transformcan still be used to align the observed 
object wth its sped he rodel. 

7 Sunnary 

Wintroduced in this paper a recognition schena that 
proceeds intwostages: c at egor i z at i on and i dentihcati on. 
Gktegorizati on is achieved by ali gni ng the i nage to pro¬ 
totype objects. lor every prototype, the nearest proto¬ 
type viewis recovered, and the si ml arity between this 
view and the inage is evaluated. The prototype that 
rost resenbles the observed obj ect deternimes its class 
identity. Iikewse, i dentihcati on i s achi eved by al i gn- 
ing the observed obj ect to the individual rodels of its 
class. A this stage the prototype transformconputed 
inthe categorization stage is reused to align the rodels 
wththe inage. The rodel that natches the observed 
object deternimes its specihc identity. In addition, we 
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El gure 2: Hctures of tw> cliairs used as mdels. We refer to these chairs bv A(left) and B(right). Mdels for the tvo cliairs 
vere constructedforming) e inages using synmtry [31]. 



El gure 3: Hctnres of tvo cars used as mdels. left: a VWmdel. light: a Saab mdel. Mdels for the tvo cars vere 
borroved from[42]. 



El gure 4: Mtcling a prototype chair (chair 4) to aninage of chair B This figure, as veil as the rest of the figures, contain 
three pictures, left: the imge to be recognized Mddle: the appearance of the prototype followng the application of the 
prototype transform Rght: an overl ay of the left and the riddle pictures. 
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presented an algorithmf or cons t r uct ing the opt iral pro¬ 
totypes and discussed the relevance of the schena tohu- 
nan recognition. 

Aiinportant issue conveyed by our schem is that 
categorization can be used to faci li tate the identification 
of objects. Wshowedthat by first categori zi ng the ob- 
j ect, the diffiult stages of the alignnant process, nanaly, 
the recovery of the object pose and the correspondence 
bet ween t he i nage and t he rodel, canbeperformd onl y 
once per cl ass. Q)nsequently, i dentihcationis reducedin 
thi s schem into a series of si nple tenpl ate conparisons. 

The schem presented in this paper differs fromex- 
isting categorization schems in two inportant aspects. 

The existing schems (e.g. , [4]) first attenpt to recover 
the part structure (geons) of the object fromthe inage 
alone. This structure is assured to be alrost invari¬ 
ant both to rotation of the obj ect and across obj ects of 
the sam class. In contrast, our schem does not at¬ 
tenpt to recover any 3 D infornation fromthe inage 
alone. Mreover, it separates the tw) effects that deter- 
mne the obj ect ’ s appearance: vi ewvari ati on effects and 
def ornati ons due to cl ass vari abi 1 i ty. \i ewvari ati ons are 
conpensatedfor by recovering the viewof the prototype 
that rost resenbles the inage, and the arount of de- 
f ornati on that separates the prototype fromthe specific 
object is evaluated by assessing the difference (in 2 D) 
between the nearest prototype viewandthe inage. 

Qen problem for future research include solving the 
correspondence between prototypes andinages, conbin- 
ingthe schem wi th existing indexing approaches, defin¬ 
ing effecti ve masures to evaluate the qualityof natches, 
and extendi ng the systemto incorporate additional cues, 
such as color and texture. 
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